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METHOD AND APPARATUS FOR REPRESENTING IMAGE GRANULARITY BY 

ONE OR MORE PARAMETERS 



5 CROSS-REFERENCE TO RELATED APPLICATIONS 

This application claims priority under 35 U.S.C. 1 19(e) to U.S. Provisional 
Patent Application Serial No 60/470,712, filed May 15, 2003, the teachings of which 
are incorporated herein. 

10 

TECHNICAL FIELD 

This invention relates to a technique for simulating film grain . 

15 BACKGROUND OF THE INVENTION 

Motion picture films comprise silver-halide crystals dispersed in an emulsion, 
which is coated in thin layers on a film base. The exposure and development of 
these crystals form the photographic image consisting of discrete tiny particles of 

20 silver. In color negatives, where the silver is chemically removed after development, 
tiny blobs of dye occur on the sites where the silver crystals form. These smalls 
specks of dye are commonly called 'grain' in color film. Grain appears randomly 
distributed on the resulting image because of the random formation of silver crystals 
on the original emulsion. Within a uniformly exposed area, some crystals develop 

25 after exposure while others do not. 

Grain varies in size and shape. The faster the film, the larger the clumps of 
silver formed and blobs of dye generated, and the more they tend to group together 
in random patterns. The grain pattern is typically known as 'granularity'. The naked 
eye cannot distinguish individual grains, which vary from 0.0002 mm to about 0.002 

30 mm. Instead, the eye resolves groups of grains, referred to as blobs. A viewer 
identifies these groups of blobs as film grain. As the image resolution becomes 
larger, the perception of the film grain becomes higher. Film grain becomes clearly 
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noticeable on cinema and high-definition images, whereas film grain progressively 
loses importance in SDTV and becomes imperceptible in smaller formats. 

Motion picture film typically contains image-dependent noise resulting either 
from the physical process of exposure and development of the photographic film or 

5 from the subsequent editing of the images. The photographic film possesses a 
characteristic quasi-random pattern, or texture, resulting from physical granularity of 
the photographic emulsion. Alternatively, a similar pattern can be simulated over 
computed-generated images in order to blend them with photographic film. In both 
cases, this image-dependent noise is referred to as grain. Quite often, moderate 

10 grain texture presents a desirable feature in motion pictures. In some instances, the 
film grain provides visual cues that facilitate the correct perception of two-dimensional 
pictures. Film grade is often varied within a single film to provide various clues as to 
time reference, point of view, etc. Many other technical and artistic uses exist for 
controlling grain texture in the motion picture industry. Therefore, preserving the 

15 grainy appearance of images throughout image processing and delivery chain has 
become a requirement in the motion picture industry. 

Several commercially available products have the capability of simulating film 
grain, often for blending a computer-generated object into natural scene. Cineon® 
from Eastman Kodak Co, Rochester New York, one of the first digital film applications 

20 to implement grain simulation, produces very realistic results for many grain types. 
However, the Cineon® application does not yield good performance for many high 
speed films because of the noticeable diagonal stripes the application produces for 
high grain size settings. Further, the Cineon® application fails to simulate grain with 
adequate fidelity when images are subject to previous processing, for example, such 

25 as when the images are copied or digitally processed. 

Another commercial product that simulates film grain is Grain Surgery™ from 
Visual Infinity Inc., which is used as a plug-in of Adobe ® After Effects ®. The Grain 
Surgery™ product appears to generate synthetic grain by filtering a set of random 
numbers. This approach suffers from disadvantage of a high computational 

30 complexity. 
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None of these past schemes solves the problem of restoring film grain in 
compressed video restoration. Film grain constitutes a high frequency quasi-random 
phenomenon that typically cannot undergo compression using conventional spatial 
and temporal methods that take advantage of redundancies in the video sequences. 
5 Attempts to process film-originated images using MPEG-2 or ITU-T/ISO H.264 
compression techniques usually result in either an unacceptably low degree of 
compression or complete loss of the grain texture. 

Thus, there exists a need for a technique for representing the film grain 
characteristics through one or more of a set of parameters. 

0 

SUMMARY OF THE INVENTION 



Briefly, in accordance with a preferred embodiment of the present principles, . 
there is provided a technique for simulating film grain. The technique commences 

15 upon the receipt of image information representative of an image from which film 

grain has been at least attenuated. Accompanying the image information is film grain 
information that includes at least one parameter among a set of possible parameters 
specifying different attributes of the film grain previously in the image. A model is 
selected for simulating grain and the film grain is simulated film grain in accordance 

20 with the selected model and the at least one parameter. The simulated film grain is 
then merged into the image. 



BRIEF DESCRIPTION OF THE DRAWINGS 



25 FIGURE 1 illustrates a block diagram of a first embodiment of a system in 

accordance with the present principles for simulating film grain in accordance with the 
present principles; 

FIGURE 2 illustrates a block diagram of a second embodiment of a system in 
accordance with the present principles for simulating film grain in accordance with the 
30 present principles; and 
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FIGURE 3 illustrates a block diagram of a third embodiment of a system in 
accordance with the present principles for simulating film grain in accordance with the 
present principles. 



5 DETAILED DESCRIPTION 

FIGURE 1 depicts a block schematic diagram of a first embodiment of a 
system 1 0 in accordance with the present principles for performing film grain 
simulation in accordance with the present principles. The system 10 includes a Film 

10 Grain Remover 22 that serves to remove the film grain from an input video stream 1 2 
to yield a filtered video stream 24 received at a Video Encoder 13. Film grain 
removal constitutes a particular case of noise filtering where the noise signal appears 
correlated with the image signal. Thus, the Film Grain Remover 22 can take the form 
of a classical image filter, although such a filter will not necessarily provide optimal 

15 performance. The Video Encoder 13 encodes the filtered video stream 24 to yield a 
coded video stream 14 for receipt at a Video Decoder 15 that decodes the coded 
stream to yield a decoded video stream 16. The Video Encoder 13 and the Video 
Decoder 15 utilize the same video coding scheme as are well known in the art. For 
example, the video coding scheme could comprise the ITU-T H.264 video-coding 

20 standard, or another type of block-based coding. Encoders and decoders that utilize 
the MPEG-2 and the ITU-T H.264 standard are well known. 

The system 10 also includes a Film Grain Characterizer 23 that receives the 
input video stream 12 and the filtered video strearh 24. From these video streams, 
the Film Grain Characterizer 23 outputs a message, hereinafter referred to as a grain 

25 message, that contains an identity of a model for simulating grain, as well at least one 
of a set of several parameters, including correlation parameters, intensity- 
independent parameters and intensity-dependent parameters used by the identified 
model. As discussed in detail below, the parameters in the film grain message 
enable simulation of the original image-dependent noise using the identified model. 

30 In the absence of any of the parameters, a default value is assigned to that missing 
parameter. (Indeed, if no model is identified, a default model for film grain is 
assumed. In one embodiment, the Film grain characterizer 23 generates the 
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parameters in accordance with a model based on the physical process of exposure 
and development of the photographic film or upon processes added during the 
subsequent editing of the images. 

Following generation of the grain message, a Film Grain characterization 

5 information Encoder 26 encodes the message for transmission to a Film Grain 
characterization information Decoder 28 in-band or out-of band from, the encoded 
video stream 14 transmitted by the Video Encoder 13 to a the Video Decoder 15. 
Both the Video Encoder 13 and the Film Grain characterization information Encoder 
26 use the same encoding scheme. Thus, for example, when the Encoder 26 utilizes 

10 the ITU-T H.264 video-coding standard for encoding, the coded film grain 
characterization information stream 27 can take the form of the film grain 
Supplemental Enhancement Information (SEI) message as defined in the ITU-T 
H.264 video coding standard. 

The Film Grain characterization information Decoder 28 decodes the coded 

15 film grain message 27 to yield a decoded film grain characterization information 

stream 29 for input to a Film Grain Restoration Processor 30. As described in detail 
hereinafter, the Processor 30 will simulate the film grain with a model identified in the 
grain message using parameters in message. In the absence of the identification of 
the model, the Processor 30 will assume a default mode. Likewise, in the absence of 

20 a specified value of a given parameter, the Processor 30 will assume a default value 
for that parameter. 

In a preferred embodiment, the grain message 25 of FIG. 1 will typically 
include one or more correlation parameters specifying Spatial Correlation, Aspect 
Ratio, Color Correlation, and Temporal Correlation. Each of these parameters is 

25 discussed below 

Spatial correlation 



30 



In an exemplary embodiment, the image-dependent correlation of the film 
grain in the spatial domain can be modeled with at least one parameter, Spatial 
Correlation. The measured spatial correlation determines the size of the blobs. A 
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second order auto regression model and a convolution model for the Spatial 
Correlation are described hereinafter. 

Aspect Ratio 

5 

Ideally, film grain should appear isotropic, with identical characteristics both in 
X and Y direction. However, in practice, the film grain actually can appear stretched 
in one direction, often resulting from factors related to film recording, such as the use 
anamorphic optics or non-square detector geometry. For that reason, when modeling 
10 film grain, an intensity-independent parameter representing the aspect ratio factor will 
complement the spatial correlation measure. The aspect ratio of the grain blobs is 
specified with at least one parameter. 

Color Correlation 

15 

In accordance with the present principles, the layer-dependency of the film 
grain in color images is represented using color correlation. The measured color 
correlation determines the perceived tint of the grain. A weak color correlation 
implies that grain blobs created in the different color layers randomly overlay each 
20 other. Consequently, a viewer will perceive the grain as colored. A high color 

correlation implies that the grain blobs on one color component depend on other color 
components. In this case, a viewer will perceive the grain as monochromatic. 

Temporal correlation 

25 

The temporal correlation of the grain in sequences is represented by at least 
one parameter. Grain by itself cannot exhibit any temporal correlation between 
frames, but the introduction of a parameter representing temporal correlation can 
help to simulate other observed effects caused by the editing of the film. 



30 
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Noise intensity 

In conjunction with the previously discussed parameters representing the 
dependency of the film grain with the film image, a need exists to represent the 

5 intensity of the noise arising from the random processes that originate the film grain. 
Noise intensity can vary for each color component and will depend on the film image. 
The intensity of the grain determines the level in which noise is perceived in the 
image. Small grain intensity levels introduce small variations in the original image 
and hardly appear noticeable. High intensity levels become clearly visible as peaks 

10 superimposed in the image. 

Other Parameters 

In addition to the parameters discussed above, the grain message can also 

15 include parameters that identify the color space in which to add the film grain, and the 
blending mode used to blend the grain with the video signal. Note that a different set 
of parameters could be transmitted for each color component and for different 
intensity levels of the film image. It is well known for example that film grain depends 
on the local intensity of the image, and that different color components can have 

20 different grain depending on the type of film stock. 

The film grain characterizer 23 of FIG. 1 can generate different sets of 
parameters in accordance with the intensity levels of the image. If desired, the Film 
grain decoder 28 can interpolate the set of parameters to various intensity levels in 
order to derive a smooth transition of the film grain characteristics. 

25 In order to interpret the set of parameters, the Film grain decoder 28 must 

have a specification of the model that generates the parameters. To understand how 
such a model can be specified, the following mathematical relationships will prove 
useful. First, the decoded image pixel value at image position (x, y), color channel c, 
and frame number t is represented by l(x, y, c, t). For convenience, assume that 

30 pixel values are scaled to have maximum value of one. Further, assume an RGB 
image representation (c = 1 , 2, or 3), although this model can be directly to 
monochromatic images and, with obvious modifications, to YUV representation. 
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With an additive grain model, grain simulation changes each pixel value to J(x, 
y, c, t) where J(x, y, c, t) is given by the relationship: 

(1 ) J(x, y, c, t) = l(x, y, c, t) + G(x, y, c, t, L(x, y, t)), 

5 

where L(x, y, t) is a measure of local intensity in the image and G(x, y, c, t, L(x, y, t)) 
defines the grain value. One possible implementation is to define L as luminance, or 
a weighted sum of intensities l(x, y, c, t) over all color channels. 

The additive model given by equation (1) is appropriate when a logarithmic 
10 intensity scale is used. For linear scale, the model of equation (1) can be replaced by 
the following multiplicative mode: 

(1 a) J(x, y, c, t) = l(x, y, c, t) * (1 + G(x, y, c, t, L(x, y, t))). 

15 Whether an additive or a multiplicative grain model is implemented will depend on the 
format of the decoded image. In general, grain should comprise small fraction of 
maximum pixel value. The following describes some examples of different kind 
of models for extracting a set of parameters in accordance with the present invention. 

20 1 . Autoregressive simulation of film grain pattern 

In an exemplary embodiment, a second order auto regression scheme can be 
used to model spatial correlation and a first order regression scheme can be used to 
model cross-color and temporal correlations. All correlation factors depend on 
25 intensity of the decoded image. Horizontal and vertical spatial correlation factors are 
related by a constant aspect ratio factor. Under such conditions, the following 
formula will yield simulated grain values, 



30 



(2) G(x,y,c,t, L) = p(c,LrN + 

q(c, L) * (G(x-1 , y, c, t, L) + A * G(x, y-1 , c, t, L)) + 
r(c, L) * A * (G(x-1 , y-1 , c, t, L) + G(x+1 , y-1 , c, t, L)) + 
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s(c, L) * (G(x-2, y, c, t, L) + A * A * G(x, y-2, c, t, L)) +u(c, L) * G(x, y, c-1 , t, L) 
v(c,L)*G(x,y,c,M,L) 

5 where N is a random value with normalized Gaussian distribution, A is a constant 
pixel aspect ratio, p, q, r, s, u, and v are correlation parameters. Parameter u is 
always zero for the first color channel, and the grain value G assumed to be zero 
whenever any index is out of range. 

As can be seen from the structure of equation (2), grain values for a given 

10 pixel in a given color channel are calculated recursively using previously calculated 
grain values. Specifically, frames are calculated in order of increasing frame number 
(i.e., increasing t). Within each frame, color channels processing occurs in order of 
increasing color channel number (i.e., increasing c). Within each color channel, 
pixels are rasterized horizontally and then vertically in order of increasing x and y. 

15 When this order is followed, all grain values required by equation (2) are 
automatically calculated in advance. 

Under certain circumstances, vertical rasterization proves more practical, i.e. 
pixels processing occurs by columns first. In such cases, equation (2) will require a 
slight modification to use only previously calculated values: 

20 

(2a) G(x, y, c, t, L) = p(c, L)*N + 
q(c, L) * (G(x-1 , y, c, t, L) + A * G(x, y-1 , c, t, L)) + 
r(c, L) * A * (G(x-1, y-1, c, t, L) + G(x-1, y+1, c, t, L)) + 
s(c, L) * (G(x-2, y, c, t, L) + A * A * G(x, y-2, c, t, L)) + 
25 u(c, L)*G(x, y,c-1,t, L) + 

v(c,L)*G(x,y,c,t-1,L). 

Implementing equation (2) or equation (2a)) requires certain minimum decoder 
capabilities. First, the Film Grain Information decoder 28 must perform all 
30 calculations real time. Second, the Film Grain Information decoder 28 needs to keep 
a number of previously calculated grain values in memory. Specifically, to implement 
temporal correlation (i.e., last term in equations (2) and (2a)), the Film Grain 
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Information decoder 28 needs to keep grain values for a full previous frame. From 
this perspective, it is important that the model of equation (2) allow gradual scaling 
down requirements with some degradation of fidelity. 

A system with slightly lower fidelity could ignore the last (temporal) term in 

5 equation (2). Doing so would eliminate the need to have an additional frame buffer to 
keep grain values from previous frame. Further cost savings would result by 
neglecting those terms in equation (2) that depend on s(c, L). Doing so eliminates 
need to store a second previous row in memory and reduces number of calculations. 
Neglecting diagonal correlations described by terms with r(c, L), and so on will 

10 achieve a further reduction of complexity. The lowest quality grain simulator will use 
only white noise term. 

Whenever a term is neglected in a scaled-down system, a benefit occurs if the 
Film Grain Information decoder 28 adjusts the remaining parameters so that the 
effective first-order correlation and even more importantly, the autocorrelation (noise 

15 power) remain the same as they would in a full-scale implementation of the model 
embodied by equation (2). The same adjustment should occur for the first rows and 
columns of each frame in the absence of the availability of all of the previous grain 
values. 

The flexibility of the model embodied in equation (2) will become further 
20 apparent by setting parameters p, q, r, and s to zero for all but first color channel, and 
by setting the color correlations u(c, L) for c> 1 to 1 . Under such conditions, the 
grain becomes completely monochromatic. This set of parameters values can 
describe the case for color variations washed out by previous YUV 4:2:0 
transformation of the color space. 
25 For a set of three colors, the model of equation (2) describes the grain pattern 

in terms of a group of seventeen parameters for each luminance level, plus an aspect 
ratio that does not depend on luminance. Luminance-dependent parameters can be 
encoded for several fixed luminance levels. The decoder will interpolate parameters 
values for intermediate luminance levels. 
30 Grain parameters do not have to be represented exactly in the form of 

equation (2). For example, any one-to-one transform of the parameters could be 
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used. In addition, different set of reference intensity levels could be used for different 
parameters and different interpolation schemes could be used, etc. 

2. Convolution in the spatial domain to simulate the film grain 
S pattern 

In another exemplary embodiment, the film grain pattern can be simulated by 
convolving a set of random numbers x by a linear, time-invariant, digital-filter h 
defined in the form: 

0 

(3) h = (ho, hi, Ii2, h3, ... h n ) 



This states that the filter output simulating film grain y(n) is the convolution of 
the input x(n) w\\b the filter impulse response h(n): 

15 

(4) y(n) = jS>(0M" " 0 = (* * h){n) 

M> 



Although equation (4) yields a simulation in one dimension, a two-dimensional 
pattern could be obtained by concatenating the vertical and horizontal convolutions in 
20 one dimension. Under such circumstances, the coefficients of the filter should be 
transmitted in addition to the aspect ratio factor. 

A Film Grain Information decoder 28 with limited capabilities can limit the 
spatial size of the convolution kernel, which will result in decreased memory and 
processing power requirements. 

25 

3. Filtering in a transformed domain to simulate the film grain pattern 



30 



As discussed previously, convolving the impulse response of a filter h with a 
set of random numbers x can characterize the film grain pattern. This same 
operation can also be described by multiplication in the frequency domain by the 
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Fourier Transform of the impulse response H and the Fourier Transform of the set of 
random numbers X: 

(5) F( M ) = X(u) •/*(«) 

5 

Filtering in the frequency domain affords an advantage because it is 
computationally faster if a Fourier transform of the image is available for example as 
part of filtering or compression process." 

The following set of parameters yielded satisfactory results to represent 
10 image-dependent grain according in accordance with the present principles. These 
parameters assume an autoregressive method of simulating grain. Parameters for 
other methods would be represented by similar tables. 

Color space: logarithmic RGB 
15 Blending mode: additive 
Aspect ratio: 1 

Number of intensity levels: 3 

Parameters for the R component: 

q r u v p 
20 level [ 0, 84] : 0.1 .01 0.0 0.2 0.02 

level [85,168]: 0.1 .01 0.0 0.15 0.03 

level [169,255]: 0.3 -.01 0.0 0.15 0.05 

Parameters for the G component: 

q r u v p 
25 level [ 0, 84] : 0.3 0.0 0.1 0.2 0.01 

level [85,168]: 0.2 .01 0.1 0.15 0.03 

level [169,255]: 0.1 -.01 0.2 0.1 0.05 

Parameters for the B component: 
30 q r u v p 

level [ 0, 84] : 0.4 .01 0.1 0.2 0.02 
level [85,168]: 0.1 0.0 0.1 0.15 0.03 
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level [169,255]: 0.1 0.0 0.2 0.1 0.04 

Correlation parameters not shown in this table are set to 0. 

After specifying the color space, the blending mode, the aspect ratio and the number 
5 of intensity levels for which different parameters are defined, the film grain on each 
color component is encoded. Note that only the data in italics in addition to some 
level (intensity) information has to be transmitted. 

FIGURE 2 depicts a second embodiment 10* of a system for simulating film 
grain in accordance with the present principles. The system 10' shares many of the 
10 same elements as the system 10 of FIG. 1 and like reference numbers describe like 
elements. Indeed, the system 10' of FIG. 2 differs only in the absence of the Film 
Grain characterization information Encoder 26 and Film Grain characterization 
information Decoder 28 of FIG. 1. The system 10* of FIG. 2 uses the Video Encoder 
13 and Video Decoder 15 to encode and decode respectively the film grain 
15 characterization information 25 output of the Film Grain Characterizer 23. The 
system 10' of FIG. 2 requires the use of a video coding standard that supports the 
transmission film grain characterization information as parallel enhancement 
information. 

FIGURE 3 depicts a third embodiment 10" of a system for simulating film grain 
20 in accordance with the present principles. The system 1 0" shares many of the same 
elements as the system 10' of FIG. 2 and like reference numbers describe like 
elements. Indeed, the system 10" of FIG. 3 differs only in the absence of the Film 
Grain Remover 22 of FIG. 2. The system 10" of FIG. 3 uses the reconstructed 
images available at the Video Encoder 13 to simulate the result of removing film 
25 grain. The system 10" of FIG. 3 affords two advantages as compared to the systems 
10 of FIG. 1 and 10' of FIG 2. First, the system 10" of FIG. 3 reduces the 
computational complexity related to film grain removal, and secondly, it adapts the 
film grain characterization to the amount of film grain suppressed by the Video 
Encoder 13. Once the Film Grain Characterizer of FIG. 3 disposes of both the input 
30 video 12 with film grain, and a reconstructed video 24 resulting from Video Encoder 
13, it can accomplish the task of characterizing the observed film grain. 
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the foregoing describes a technique for simulating film grain in a video signal. 
While the film grain simulation technique has been described in connection with the 
encoding and decoding of a video signal, the technique has equal applicability for 
other purposes, such as for example post-production of motion picture films for 
5 example. In this regard, the original image could exist as image information in a form 
other than a compressed video signal, and the film grain information could exist in a 
form other than as a message, such as an SEI message. For example, the image 
information could exist in one of a variety of different formats that exist in the art. 



